The Vector Space Model in Information Retrieval - Term Weighting Problem

Many traditional information retrieval (IR) tasks, such as text search, text clustering or text categorization, have natural language documents as their first-class objects, in the sense that the algorithms that are meant to solve these tasks require explicit internal representations of the documents they need to deal with. In IR documents are usually given as extensional vectorial representation, in which the dimensions (features) of the vector representing a document are the terms occurring in the document. The approach to term representation that the IR community has almost universally adopted is known as the bag-of-words approach: a document dj is represented as a vector of term weights −→ dj = 〈ω1j , ..., ωrj〉, where r is the cardinality of the dictionary and 0 ≤ ωkj ≤ 1 represents the contribution of term tk to the specification of the semantics of dj . This article analyses and compares many different bag-of-words approaches.